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This study investigated how patterns of school violence change across years using the 
data extracted from the school questionnaire in TIMSS and TIMSS-R. Violence was 
operationally defined as a continuum or hierarchy of physical and nonphysical aggression. 
The study found that four types of violence ( intimidation or verbal abuse of other students, 
intimidation or verbal abuse of teachers or staff physical injury to other students, and 
physical injury to teachers or staff) were reported with decreasing frequency and two 
( alcohol use/possession and illegal drug use/possession) were reported with increasing 
frequency for the eighth graders for 1995 than for the eighth graders in 1999. Alcohol 
use/possession and illegal drug use/possession became increasingly an issue that should be 
addressed by the schools and the family. Going from grade 4 to grade 8, vandalism was 
reported more frequently while physical injury to other students was reported less frequently. 
The severity of violent behaviors perceived was related to the frequency they were reported. 
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Background 

School violence is a controversial and complex issue. Yu (2001) noted that published 
reports and studies portrayed a mixed picture of school safety and that the lack of consensus 
on what is defined as school violence makes comparisons of findings across studies difficult. 
Small and Tetrick (2001) also found it difficult to depict a thorough picture of school 
violence, due to the lack of common indicators of violence, unavailability of data from grade 
levels other than high school, and different definitions for incidents. 

To investigate the nature of school violence and its patterns across elementary, junior 
high, and high schools, Yu (2001) utilized school questionnaire data collected in the Third 
International Mathematics and Science Study (TIMSS) and extracted the items from the 
school principals’ responses as indicators of school violence. She found that school violence 
consisted of a hierarchy of violent behaviors with increasing intensity from the bottom to the 
top. She also found that across all grade levels the most frequently reported type of violence 
is intimidation or verbal abuse of students and the least frequently reported is physical injury 
to teachers or staff . She did not find evidence showing that violence was the norm in the U.S. 
schools. 

The Third International Mathematics and Science Study was the largest and most 
complex study that assessed student achievement in mathematics and science at third and 
fourth grades, seventh and eighth grades, and the final year of school. Conducted in 1995, 
TIMSS also gathered background information at the student, the teacher, and the school level 
to understand the context under which learning and teaching took place. To measure trends in 
student mathematics and science achievement since 1995, TIMSS assessed again in 1999 

(known as TIMSS-Repeat, or TIMSS-R) the mathematics and science achievement of the 
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eighth-grade students and collected extensive background information from students, 
teachers, and school principals. TIMSS-R, similar to TIMSS, used a two-stage sampling 
procedure to ensure that the sampled students were representative of the national eighth 
grade student population (Foy & Joncas, 2000). In the first stage, schools were randomly 
selected, and in the second stage, classrooms were randomly selected within schools. 
Approximately 150 schools were randomly selected from each participating country for the 
study. Twenty-six of the forty-two countries that participated in TIMSS also participated in 
TIMSS-R (Martin & Mullis, 2000). U.S. participation in both TIMSS and TIMSS-R and the 
availability of school questionnaire data made it possible to examine the trends of school 
violence across time. 

Research Questions 

This study was designed to investigate how patterns of school violence change across 
years and used the same theoretical base and methodology, as were used in Yu (2001). 
Violence was operationally defined as a continuum or hierarchy of physical and nonphysical 
aggression. 

Eighth grade students in 1995 were assessed in TIMSS and eight grade students in 
1999 were assessed in TIMSS-R. In addition, students in fourth grade in 1995 (TIMSS) were 
in eighth grade in 1999 (TIMSS-R). This study addressed the following research questions: 

1 . What was the trend of school violence between the eighth grade in 1995 and the 
eighth grade in 1999? 

2. What were the changes in the reported patterns of school violence from the fourth 
grade to the eighth grade? 
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3. What was the relationship between the severity and the frequency of school 
violence? 

Method 

Sample. The samples used in the study included all the U.S. schools that contained 
eighth grade students selected to participate in TIMSS and TIMSS-R. All the schools that 
contained fourth grade students selected to participate in TIMSS were also included. As was 
mentioned above, these schools were randomly selected according to the sampling design of 



TIMSS and TIMSS-R. 



Instrument. The same items were extracted from TIMSS and TIMSS-R as indicators 
of school violence. They were vandalism, theft, intimidation or verbal abuse of other 
students, physical injury to other students, intimidation or verbal abuse of teachers or staff 
physical injury to teachers or staff alcohol use/possession, illegal drug use/possession, 
weapon use/possession, and inappropriate sexual behavior. Data on these ten items came 
from school administrators’ responses to the question "About how often does the school 
administration or staff have to deal with following behaviors among Grade 4/Grade 8/ 1 
students?" in the school background questionnaire. As only the first six items were 
administered to Grade 4 schools, data on these items only were used involving the fourth- 
grade analysis. 

There was one notable change in the rating scale categories used in TIMSS and 
TIMSS-R. In TIMSS, the rating scale has four response options - “rarely”, “monthly”, 
“weekly”, and “daily”, while in TIMSS-R, the rating scale has five response options - 



1 Only one grade level was used in questionnaires addressed to the administration of the schools that 
contained that grade level. 
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“never”, “rarely”, “monthly”, “weekly”, and “daily”. At the same time the school 
administrators were asked about the frequency with which they dealt with these problems, 
they were asked to rate their severity: “To what extent do these behaviors present a problem 
in your school?” The scale provided for rating the severity of the problems has three options: 
“not a problem”, “minor problem”, and “serious problem”. 

Analysis. The study was conducted using the Rasch rating scale model (Wright & 
Masters, 1982). The use of the rating scale model assumes that all the items share the same 
category structure. Free calibrations centering on each sample were conducted to examine the 
trends of school violence both across years and across grade levels. As the rating scale used 
in TIMSS-R is different from that used in TIMSS, the category options in TIMSS-R were 
collapsed to conform to those used in TIMSS. That is, the two categories -“never” and 
“rarely” — were combined to form one category called “rarely.” In the comparison of 
frequency and severity ratings, anchored person estimates from the frequency analysis were 
used in the severity analysis so that the items estimates obtained were on the same scale. For 
easy interpretation, the mean of the scale was set to 50, and 1 logit was rescaled to be 9.1 so 
that the scale ranged from 0 to 100 with increments of 10 units, as was used by Yu (2001). 
Results 



School demographics show that the same types of schools that contained eighth grade 
students were selected in both TIMSS (N=183) and TIMSS-R (N=240), with similar 
percentages of schools from geographically isolated areas and rural areas. However, more 
schools close to the center of a town/city and fewer schools on the outskirts of a town/city 
were sampled in TIMSS-R than in TIMSS. Schools had similar enrollment of boys and girls 
from eighth grade in both studies. 
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Insert Table 1 About Here 

Rasch analysis was conducted using Winsteps 2.98 (Linacre & Wright, 1999). 
Separate calibrations were run for 1995 and 1999 data for the eighth grade, with the mean of 
each sample set to 50. The results show that the person separation reliability and the person 
separation index were .81 and 2.05 for 1995, and .86 and 2.44 for 1999, respectively. The 
item separation reliability and the item separation index were .97 and 5.87 for 1995, and .99 
and 10.45 for 1999, respectively. Fit statistics show that physical injury to teachers or staff 
and inappropriate sexual behavior were misfitting items for 1995 while only inappropriate 
sexual behavior was a misfitting item for 1999. Both person and item statistics improved 
from 1995 to 1999 data. Across the time the most frequently reported type of violence was 
intimidation or verbal abuse of other students and the least frequently reported type of 
violence was physical injury to teachers or staff although the ordering of other items in 
between changed. T-tests (Wright & Masters, 1982) were conducted to compare the 
measures obtained for 1995 and 1999. The results show that there were statistically 
significant changes between the two sets of measures for some items (t >1 .96, p < .05). Four 
items, intimidation or verbal abuse of other students, intimidation or verbal abuse of 
teachers or staff and physical injury to other student and physical injury to teachers or other 
staff were reported significantly less frequently in 1999 than in 1995 while two items, 
alcohol use/possession and illegal drug use/possession, were reported significantly more 
frequently in 1999 than in 1995 for the eighth grade. The other four items remain invariant 
across the years. 

Changes in item estimates for grade 4 in 1995 going to grade 8 in 1999 were also 

examined. As only six items were administered to administrators of schools that contained 
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grade 4 in 1995, the same number of items was extracted for analyzing grade 8 data in 1999. 
There were no misfitting items for the analysis of grade 4 data. A t-test shows that vandalism 
was reported more frequently with grade 8 than grade 4 (t=2.58) while physical injury to 
other students was reported more frequently with grade 4 than with grade 8 (t=-3.89). 

Besides reporting the frequency with which they dealt with problem behaviors, school 
principals were also asked simultaneously to rate the seriousness of each problem. To 
examine the relationship between frequency and seriousness of each problem, the results 
from frequency analysis were compared to those from severity analysis. Person estimates 
from the frequency analysis were anchored in the severity analysis so that item estimates 
from the severity analysis were on the scale of the frequency analysis. The correlation 
between the two sets of item statistics was .96, indicating that the severity of violent 
behaviors were strongly correlated with the frequencies with which they were reported. Item 
estimates from the two analyses were placed in Figure 1. The graph shows that the more 
frequent a violent act is reported, the more serious it is reported, although frequency and 
severity may not be at the comparable level. Illegal drug use/possession, weapon 
use/possession, alcohol use/possession and inappropriate sexual behavior were rated 
comparably in terms of frequency and severity. Other five items, intimidation or verbal 
abuse of other students, vandalism, theft, physical injury to other students, and intimidation 
or verbal abuse of teachers or were reported in higher level of frequencies than they were 
perceived with the degree of severity. 
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Discussion 

Trends of school violence for the eighth grade population between 1995 and 1999 
were found in this study. Four out of the ten types of violence (i.e., vandalism, theft, weapon 
use/possession, and inappropriate sexual behavior) did not show any difference in their 
reported frequency. Four types of violence (i.e., intimidation or verbal abuse of other 
students, intimidation or verbal abuse of teachers or staff, physical injury to other students, 
and physical injury to teachers or staff) were reported with decreasing frequency and only 
two ( alcohol use/possession and illegal drug use/possession) were reported with increasing 
frequency. Recall that Yu (2001) reported that alcohol use/possession and illegal drug 
use/possession were reported more frequently for grade 8 than for grade 12 in TIMSS. And 
they were reported even more frequently four years later. Therefore, alcohol use/possession 
and illegal drug use/possession, replacing interpersonal conflicts, became increasingly an 
issue that should be addressed by the schools and the family. 

Going from grade 4 to grade 8, vandalism was reported more frequently while 
physical injury to other students was reported less frequently. The severity of violent 
behaviors perceived was related to the frequency they were reported. In general, the more 
frequent a violent behavior is reported, the more serious it was perceived. 

School violence is a complicated phenomenon. Research on school violence faces a 
lot of challenges due to the lack of comprehensive data, appropriate data gathering methods, 
denial, and underreporting of violence data. The data gathered from the national 
representative samples of schools from both TIMSS and TIMSS-R made it possible to 
investigate the trends and patterns of school violence across time. The use of Rasch modeling 
provides much more informative information about the data and its quality than traditional 
analysis. The fact that the school administrators reported the frequency with which they dealt 
with these problems and rated their severity at the same time made it possible to examine the 
relationship of the two for the first time. The findings of this study contribute to school 
violence literature. 
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Table 1. 



Demographics of the schools with eighth grade students 




TMSS 


TIMSS-R 


Type of community 


A geographically isolated area 


2.6% 


3.8% 


Village or rural (farm) area 


18.1% 


17.0% 


One on the outskirts of a town/city 


29.0% 


23.1% 


One close to the center of a town/city 


50.3% 


56.0% 


Total number of schools 


183 


240 


Average of total school enrollment 




Boys 


396 388 




Girls 


364 376 
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Frequency and Severity 
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Figure 1. Comparisons of frequency and severity estimates Using TIMSS-R. 
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Abstract 

This paper presents a study associated with a large-scale evaluation conducted to assess the 
impact of a research-based model entitle Second Step (SS) on primary and cognitive school 
indicators. In a partnership between a community-based organizations and a large urban public 
school district, SS provided services to at-risk students. The theory of violence prevention and 
educational accountability framed this research project. Primary data indicated positive impact of 
the program on students’ prosocial behavior. The evaluation revealed no effect of SS on 
academic indicators two years after the implementation of the program. Implications for practice 
and future research are discussed. 

Keywords: Partnerships in Education; Urban Schools; Accountability; Violence Prevention; 

High Risk Students; Early Intervention; Program Effectiveness; Program Evaluation. 
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Evaluating the Impact of Second Step on Psychosocial and Cognitive Measures 

The violence prevention literature informed the development of the stated objective 
measures. Currently, there is a call for more use of objective outcome measures (Webster, 1993). 
Several studies report improvements in knowledge and attitudes related to use of violence to 
resolve conflicts (e.g., Aber, Jones, Brown, Chaudry, & Samples, 1998; Hausman, Spivak, & 
Prothrow-Stith, 1995; Oprinas, Parcel, McAlister, & Frankoweski, 1995), or self-reported 
aggressive behaviors (e.g., Avery-Leaf, Cascardi, O’Leary, & Cano, 1997; DuRant, Barkin, & 
Krowchuk, 2001; DuRant, Treiber, Getts, McCloud, Linder, & Woods, 1996). 

Fewer studies have examined changes in relevant student behavioral indicators, such as 
suspension rates (Farrell, Meyer, & White, 2001; Housman et al., 1995) and disciplinary referrals 
(Farrell et al, 2001; Twemlow, Fonagy, Sacco, Gies, Evans, & Ewbank, 2001). Only one study 
(Farral et ah, 2001) has examined the link between changes in knowledge and attitudes and 
relevant behavioral indicators, with the results indicating no mediating effects. Thus, it is 
becoming increasingly important to examine the effectiveness of violence prevention 
interventions on student behavior and performance. 

Research studies have demonstrated that there is continuity in aggressive behavior over time: 
children who have aggressive behavior in the elementary school years are more likely to display 
antisocial and violent behaviors as adolescents and young adults (Farrington, 1991; Tremblay et 
ah, 1992). In this regard, early intervention has been advocated as most appropriate to break this 
chain of events (Tremblay & Craig, 1995; Yoshikawa, 1994). In this regard, school "violence" is 
a continuum of behavior within a developmental framework. For example, violent behavior for 
young elementary school children primarily consists of aggressive behaviors such as kicking, 
hitting, spitting, or name calling. As children grow older, behavior becomes more serious, 
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characterized by bullying, extortion, and physical fighting. Aggressive or violent adolescents 
may engage in assault against other students and staff, sexual harassment, gang activity, or 
weapon carrying. School violence is also relevant. It has been defined as conflict between 
students and teachers (Curcio & First, 1993), or as activities that cause suspensions and 
disciplinary contacts or detentions. Studies of school violence have variously used such terms as 
aggression, conflict, delinquency, conduct disorders, criminal behavior, antisocial behavior, and 
violence, among others, to describe this class of problem behaviors. Aggressive behavior is 
different from violence and antisocial behavior. 

To understand risk for violence along a developmental continuum, and to provide a 
framework for school-based prevention and intervention efforts, it is essential to understand risk 
factors for aggression and violence. It is also essential to understand the protective factors that 
schools can foster or provide to reduce a child's risk of engaging in or being victimized by 
violence. According to the American Psychological Association (APA, 1993) the preponderance 
of evidence suggests that violence is learned behavior. This does not mean that physiological or 
temperamental factors are unrelated to the manifestation of aggressive or violent behavior, but 
that, for most individuals, violence is learned behavior. This has tremendous implications for 
understanding risk factors and related attempts at prevention and intervention. 

In a review of comprehensive strategies for dealing with adolescent problem behaviors, 
Wilson and Howell (1995) identified four broad categories of risk factors: (a) individual 
characteristics, (b) family and school influences, (c) peer group influences, and (d) neighborhood 
and community effects. They also outlined three broad categories of protective factors that may 
be instrumental in moderating an adolescent's exposure to risk for delinquency involvement: (a) 
individual characteristics (e.g., resilient temperament and prosocial attitudes); (b) close affective 
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ties with family members, teachers, and friends; and (c) healthy beliefs as well as clear standards 
for behavior. 

Recognizing violent behavior as a complex phenomenon that is manifested in many different 
ways, the focus here is to inform the development and implementation of school-based violence 
prevention strategies. The list is not meant to be exhaustive. The categories of risk are: cognitive 
abilities and factors influencing school achievement, the stability and early onset of aggressive 
behavior, family factors, and the influence of media on aggressive and violent behavior. Several 
excellent reviews examine risk factors for aggression and violence (Earls, 1994; Elliot, 1994; 
Farrington, 1991; Fraser, 1996; Loeber, 1990; Reiss & Roth, 1993; Yoshikawa, 1994). 

One of the most consistent findings in the risk factor literature is that it is possible to predict 
with a high degree of accuracy which children will be aggressive and violent in adolescence by 
their behavior in kindergarten and first grade (Farrington, 1991; Loeber & Hay, 1994). The more 
serious, and the greater the variety and frequency of early aggressive behavior, the greater the 
risk of antisocial and criminal behavior continuing into later adolescence and adulthood 
(Blumstein, 1995). Generally researchers agree that early conduct problems in kindergarten and 
first grade lead to poor school achievement in later grades which, in turn, leads to delinquency in 
adolescence (Hawkins, Von Cleve, & Catalano, 1991; Hawkins et al., 1992; Tremblay et al., 
1992). 

Other longitudinal work illustrates the tremendous stability of aggressive behavior (Eron & 
Huesmann, 1993). While not all children who are identified as aggressive in elementary school 
grow up to become delinquent adolescents and violent adults, the majority of delinquent 
adolescents and violent adults retrospectively would have been able to be identified early in 
childhood as having significant behavior problems (Tolan et al., 1995). 
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The phenomenon of early onset problem behavior and its stability and chronicity illustrate 
the importance of early prevention and intervention, especially given the complex interplay of 
factors that become more difficult to ameliorate as children get older, such as ongoing school 
failure, peer rejection, persistent conflicts with teachers, and affiliation with other at-risk peers 
(Constantino, 1995). The window of opportunity for effective prevention and intervention 
diminishes as children mature. 

Accountability and School Indicators 

School systems need quantifiable measures of student performance effectiveness in a 
high-stakes accountability environment. In this environment, educational policy making is based 
on objective information, and although no single means of data collection is sufficient, the data 
generated by well-designed program evaluations are crucial to an understanding of project 
impact. Policy-makers have to refocus the educational reform efforts in general, and the 
educational excellence issues in specific, toward results on school-related indicators (Munoz, 
2002 ). 

School districts must guarantee that programs have a demonstrably positive effect on 
students’ key cognitive indicators. Although many of the interventions implemented through SS 
have produced favorable results in previous efficacy evaluations, these evaluations have focused 
mainly on outcomes related to indirect measures of behavior, such as changes in attitudes, 
knowledge, and self-reported delinquency (e.g., Cooper, Lutenbacher, & Faccia, 2000; Mytton, 
DiGuiseppi, Gough, Taylor, & Logan, 2002). School administrators are far more interested in 
outcome results directly related to student behavioral problems. Under the conceptualization of 
accountability as performance, output educational indicators are used to track and evaluate 
program effectiveness based on student results. Accountability is generally conceived as a 
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demand to judge school programs by their outputs. Accountability systems have been designed 
to track the progress of educational reforms. The function of an accountability system in 
education is to monitor and evaluate the performance of the educational programs (Wholstetter, 
1991). Schools are expected to make wise use of public resources not only by efficient cost 
accounting procedures but also by increasing attendance and decreasing suspensions. 

From a purely theoretical perspective, Murphy (1988) analysis on the relationship between 
equity and excellence is relevant in this study. It is this conceptualization that integrates the 
principles of equity and excellence an important issue for the educational reform efforts in an 
accountability era. The third-generation conceptualization of equity basically comprehends 
equity as student opportunity to leam; the first-generation only focused on equity as access (i.e. 
input) and the second-generation focused on equity as school activities and processes. In this 
regard, this conceptualization goes beyond the traditional input and process focus of prior 
educational reform efforts and establishes an interesting link with the school efforts toward 
quality expressed in terms of student achievement. Significant policy changes have to be framed 
by the conceptualization of equity as excellence in the accountability educational reform era. In 
this regard, this conceptualization of equity is highly inter-related to accountability understood as 
performance. Under the conceptualization of accountability as performance, output educational 
indicators are used to track and evaluate schools. 

The move toward greater accountability in education has been one of the hallmarks since the 
1970s in public education (Rich, 1985). Since 1974, Levin has argued that performance 
accountability is concerned with educational outputs. Levin (1974) defined performance 
accountability as “a periodic report of the attainments of schools and other educational units” (p. 
364). Performance reporting includes such measurement techniques as statewide assessments, 
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school report cards, and performance indicators. The overall objective of a performance 
accountability system is to provide a standard upon which a school can compare its own progress 
over time. The end results should (a) stimulate actions to improve education, (b) monitor 
regulatory compliance for state requirements, and (c) produce rewards as well as sanctions to 
schools (Kirst, 1990). 

Since the 1980s, education has been “rediscovered” and carefully examined. According to 
Nelson, Palonsky, and Carlson (1990), the schools needed again a reform. Previous generations 
of education reformers were concerned with making education available to the children of all 
classes and races. Instead of availability, the 1980s generation was now forced to consider the 
quality of school experiences. As Adler (1982) argues, the legal mandate for education cannot be 
satisfy only by guaranteeing all children access to education. To satisfy the educational 
responsibilities of a democratic society, public education must demonstrate that each student is 
provided with adequate levels of knowledge and skills. According to Nelson, Palonsky, and 
Carlson (1990), educational outcomes cannot longer be measured only in quantity (e.g., years of 
schooling and the number of high school diplomas granted). Schools must guarantee that 
education has a demonstrably positive effect on students. 

Schools must show that students benefit from their years of attendance, that increased 
investment in schooling can be measured in greater ability to read, write, and do 
mathematics, and that moving up the academic ladder from grade to grade is based on merit 
rather than on social promotion. (Nelson, Palonsky, & Carlson, 1990, p. 286) 

The members of the accountability movement believe that answers to qualitative questions 
must be based on hard data. Schools need quantifiable measures of student performance and 
teacher effectiveness if accountability was to be implemented. Intelligent policy decisions should 
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be based on objective information, and although no single means of data collection is sufficient, 
the data generated by well-designed school indicators are crucial to an understanding of school 
outcomes. Impressionistic data was not sufficient and anecdotal data was not scientific. The 
accountability era had entered into public education and was here to stay until today. 

Brief Description of the Second Step Program 

The selected research-based interventions and best practices under evaluation fall into three 
categories (Tolan, Guerra, & Kendall, 1995): universal interventions (those in which 
participation is a consequence of class attendance and that target students broadly), selective 
interventions (targeting specific groups within a population that have elevated risk for 
developing a problem), and (3) indicated interventions (targeting high-risk members of a 
population that are exhibiting detectable signs or symptoms of developing a problem). The 
universal interventions evaluated here is SS. 

SS is a research-based violence prevention program for K-middle school aged children. SS is 
designed to prevent aggressive behavior by increasing prosocial behavior. Prosocial behavior 
reflects competence in peer interactions and friendships and in interpersonal conflict resolution 
skills. According to Grossman and colleagues (1997), the SS violence prevention curriculum 
appears to lead to a moderate observed decrease in physically aggressive behavior and an 
increase in neutral and prosocial behavior in school. 

The objective of SS is to increase children’s ability to identify what others are feeling, take 
others’ perspectives, and respond emphatically with others. The program has also the objective 
of decreasing impulsive, aggressive, and angry behavior. SS has 28 lessons each school year. 

The focuses of the lessons are on precursor behaviors that are incompatible with violence such as 
(a) empathy, (b) impulse control, (c) problem solving, and (d) anger management. Lessons are 
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developmental^ appropriate in content and delivery with ample opportunity for students to 
model, practice, and reinforce their pro-social behavior. 

SS is designed to prevent aggressive behavior by increasing prosocial behavior, reflected by 
competence in peer interactions and in interpersonal conflict resolution skills. Based on the 
"habits of thought" model that violence can be unlearned, SS includes activities to help students 
acquire empathy, impulse control, problem-solving, and anger management skills. 

A relatively recent comprehensive and well-designed evaluation of the SS program showed 
that 2 weeks after the 30-lesson curriculum, students in the intervention group were rated by 
behavioral observers to be less physically aggressive and to engage in more neutral/positive 
behaviors on the playground and in the lunchroom (but not in the classroom) than students in the 
control group. Some of the changes persisted at 6 months post-intervention, although neither 
teachers nor parents rated significant behavior change (Grossman et al, 1997). 

This paper will inform school administrators about the impact of the program on key 
performance indicators associated with the school environment. The present study was designed 
to contribute to existing theoretical and practical knowledge about the effectiveness of a family 
of research-based and best practices interventions on school-related research issues not 
previously addressed in the literature. According to Chen and Rossi (1983), the use of theoretical 
models in program impact assessment can heighten the power of experimental designs and 
compensate for some deficiencies in the quasi-experimental designs. In the language of Rossi, 
Freeman, and Lipsey (1999), the distal outcome of improving student academic scores reflects a 
conceptual hypothesis of the implicit program impact theory. 

The outcome evaluation was on primary and secondary measures. The primary evaluation 
includes two cohorts and utilizes random selection procedures. The outcome using secondary 
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measures includes only the first cohort of students who participated in SS interventions during 
the course of the 2000-2001 school year and that standardized test scores were available after 
two years of program implementation (school year 2002-2003). The specific research questions 
that guided the evaluation are outlined below. 

Primary Outcome Questions 

Question P-1 : Is there a statistically significant difference between pre- and posttest on 

knowledge and skills associated with prosocial behavior in each of the schools? 

Question P-2: Is there a statistically significant gain between pre- and posttest on knowledge and 
skills for all the participating schools of district? 

Secondary Outcome Questions 

Question O-l. Is there an improvement in cognitive outcomes (reading, language arts, and 
mathematics) as a result of participation in SS in comparison to a matched 
control group? 

All programs being evaluated began full operation in the fall semester of 2000. Each of these 
interventions was evaluated using the methodology described more fully in the next section. 

Method 

The methodology presented in this section of the study was used to address specific research 
questions concerning one of program stated goals: To positively impact the well-being of 
students. Outcome evaluation is necessary in order to determine whether an intervention has the 
desired effect on relevant criteria by which it is to be judged (Rossi, Freeman, & Lipsey, 1999). 

The primary outcome evaluation used a randomly selected sample of school years 2000-2001 
(Cohort 1) and 2001-2002 (Cohort 2). A single group pre-posttest design was used. The 
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secondary outcome evaluation employed a pre-post matched control group design. Due to the 
inability to randomly assign individuals to intervention and control groups, a two-level matching 
procedure was employed in order to add rigor to the internal validity (Cook and Campbell, 1979) 
of the evaluation. The first level involved the selection of a subset of schools of the district in 
which SS was not being implemented but which were similar in global characteristics (e.g., 
percentage of students involved in the free and reduced lunch program, percentage of students in 
special education programs, percentage of students suspended) to Project schools. These schools 
served as the basis for the second level of matching which took place at the individual level. The 
control group “partners” for each of the intervention group individuals were selected from the 
subset of control schools. Students were matched on three demographic characteristics 
considered important by key district-level administrators: (1) Exceptional Childhood Education 
(ECE) status, (2) gender, (3) age, and (4) free-reduced price lunch. This procedure resulted in 
providing a unique, matched control group for the cohort of students receiving SS services. 

Participants 

The primary data outcome evaluation included randomly selected students that took both the 
pre- and the posttest in Cohort 1 (N = 168) and Cohort 2 (N = 225). All students were on first 
grade of the participating elementary schools. Table 1 presents the socio-demographic 
characteristics of these students. Given the high negative correlation between free/reduced lunch 
and single parent family structure with academic achievement, it can be concluded that these 
students were academically at-risk (Munoz & Dossett, 2001). 
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Table 1 

Profile of Participating Students 





COHORT 1 (N= 168) 




Race 


Gender 


Lunch Status 


Family Structure 


55.9% Minority 


52.1% Female 


79.8% Free/Reduced 


77.3% Single 


44.1% White 


48.9% Male 


20.2% Pay 


22.7% Dual 




COHORT 2 (N = 225) 




Race 


Gender 


Lunch Status 


Family Structure 


59.5% Minority 


54.1% Female 


81.5% Free/Reduced 


73.2% Single 


40.5% White 


45.9% Male 


18.5% Pay 


26.8% Dual 



For the secondary outcome evaluation, the matching was implemented using students from 
matched control schools along four dimensions: ECE status, gender, free lunch status, and age. 
The study included 922 SS students and 918 control students. The matching success rate, on 



average, was 96%. One-way analysis of variance were used to determine whether the two groups 
of students statistically differed across the matching variables. As presented in Table 2, the 
distribution of students in the control groups was very similar to the students in the intervention. 

Table 2 



Results of the computerized matching procedure for Study Participants 







ECE Status 






Gender 




Intervention 


None 


Mental / 
Physical 


EBD 


Female 


Male 


Age 

(in years) 


SS 

Control 


733 (80%) 
733 (80%) 


177(19%) 
175 (19%) 


12 (1%) 
10(1%) 


448 (49%) 
446 (49%) 


474 (51%) 
472 (51%) 


8.29 

8.30 
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Instrumentation 

Primary Outcome Data. In general, quantitative measures will be based on already established 
data collection mechanism of the county under examination. Data will come from the program 
director and from the Management Information System (MIS) of the county. The Evaluation 
Interview for SS was used to measure the students in the primary program of the school district 
under study. The purpose of the Evaluation Interview is to assess the degree of knowledge and/or 
skills a student has before and after the intervention. Photos are placed one at a time on a table or 
desk with the student sitting opposite of the interviewer. The procedure is standardized and 
includes (a) consistency, (b) reading the questions as written, (c) pacing, (d) probes, and (e) 
recording answers. The instrument has established validity and reliability. Raw scores are 
recorded in the instrument. This measure was used as the outcome criteria for establishing 
success of the program at the school level. 

Secondary Outcome Data. The. outcome evaluation relied on secondary data collected by school 
personnel and maintained in the school systems’ Management Information System using 
Teradata database software. Cognitive measures were the Comprehensive Test of Basic Skills 
(CTBS). The primary dependent variable used in this achievement study was the CTBS Normal 
Curve Equivalent (NCE) scores in reading (Kramer, Conoley, & Murphy, 1992). NCE scores 
ranges from 1 to 99 with an average of 50 and a standard deviation of 21; these scores compare 
the students’ performance to a national norm group. The CTBS is a standardized achievement 
test that was group-administered at the end of the school year and only to third grade students. 
The CTBS includes reading, language arts, and mathematics subtests. The Level 13 test is only 
administered to the third graders and has 30 multiple-choice items in reading. This study was 
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conducted only with students who had a CTBS score when in third grade in the school year 
2002-2003 (i.e., two years passed after the intervention). 

In addition to the outcome measures just described, demographic characteristics of the 
students were also collected. These included Exceptional Childhood Education (ECE) Status (a 
general indicator of the student’s cognitive, psycho-social and physical functioning), gender, and 
age (chronological age in years). Data on free-and-reduced lunch status (in which students are 
classified according to their ability to pay for their own school meals, ranging from those who 
receive free meals to those who pay full price for meals) were also collected. These measures 
were examined to assess the degree to which students in the intervention groups were matched to 
control students. 

Analysis Strategy 

The primary data analyses used dependent-sample t-tests. The secondary outcome data 
were first examined to test the statistical assumptions (e.g., distributional assumptions of the 
outcomes, homogeneity of variance, examination of outliers) of the planned analysis procedures 
(Tabachnik & Fidell, 1996). Since the data were found to be amenable to general linear 
modeling, the interventions were evaluated using a multivariate analysis of variance 
(MANOVA) with group (intervention group vs. control group) as a between-subjects factor, and 
multiple cognitive measures as dependent variables. Students were matched on ECE status, 
gender, free-reduced prices lunch, and age (see results below). Covariates were not needed to 
rule out the effects of potentially confounding factors on assessing the change in cognitive 




outcomes. 
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Primary Outcome Results 



Results 



Statistically significant differences were found in the pre- and posttest analysis at the 
district and at the school level. Table 3 shows the pre-test and posttest measures and their 
statistically significant t-value at each of the participating schools at the alpha level .001. 
Table 3 



Elementary Schools Participating in SS 



COHORT 1 COHORT 2 



Schools 


N 


M Pre M Post 


t-Ratio 


N 


M Pre M Post 


t-Ratio 






(SD) (SD) 






(SD) (SD) 




School A 


7 


15.14 25.00 


4.48* 


26 


223.81 28.38 


3.55* 


School B 


14 


(3.33) (5.44) 
17.14 19.50 


2.47* 


15 


(4.62) (6.09) 
22.47 34.67 


6.90* 


School C 


8 


(3.95) (1.28) 
16.12 26.00 


9.63* 


13 


(2.80) (7.40) 
25.54 34.46 


3.24* 


School D 


16 


(2.29) (2.13) 
14.56 22.12 


6.53* 


24 


(2.90) (8.95) 
23.29 28.46 


2.94* 


School E 


9 


(2.25) (4.34) 
19.88 25.44 


1.81 


13 


(3.77) (7.22) 
26.85 33.62 


4.01* 


School F 


20 


(8.16) (5.65) 
15.45 24.70 


9.96* 


25 


(3.33) (5.33) 
26.96 27.60 


.37 


School G 


15 


(2.74) (3.62) 
14.60 21.73 


6.79* 


21 


(6.84) (4.74) 
24.33 24.71 


.29 


School H 


11 


(3.26) (2.12) 
14.72 26.54 


8.14* 


23 


(2.95) (3.78) 
23.61 27.30 


2.61* 


School I 


12 


(3.22) (4.63) 
14.66 25.83 


5.76* 


13 


(4.28) (4.19) 
24.23 30.46 


4.36* 


School J 


15 


(5.95) (3.27) 
13.60 24.66 


12.96* 


20 


(3.53) (5.41) 
12.20 16.50 


4.67* 


School K 


14 


(3.26) (2.69) 
13.64 22.21 


7.84* 


16 


(3.08) (2.21) 
23.56 34.19 


4.38* 


School L 


27 


(2.23) (3.51) 
15.78 20.59 


9.24* 


16 


(3.18) (8.27) 
22.81 23.13 


.29 






(2.63) (2.27) 






(2.51) (2.96) 




District 


168 15.27 23.20 


19.70* 


225 


24.30 29.15 


8.55* 






(3.82) (3.99) 






(4.27) (6.85) 
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Secondary Cognitive Outcome Results 

No statistically significant differences were found between treatment and control group 
on gender (F= .18,/? = .67), ECE category (F= 1.35,/? = .25), free-reduced price lunch (F = 
1.53,/? = .22), and age (F= .01,/? = .91). This confirmed that the groups were equivalent. 
Results of the cognitive study indicated no statistically significant gains for the SS group when 
compared to the control group. In fact, the control group outperformed the treatment group in 
both reading and language arts. Table 6 displays the results associated with cognitive measures. 



Table 6 

Results of intervention impact on Cognitive outcomes. 



Cohort 1 Outcomes 


SS (n = 
M 


703) 

SD 


Control (n = 422) 
M SD 


F 


Eta 


Reading 


44.28 


20.41 


47.48 19.96 


6.56* 


.08 


Language Arts 


44.51 


18.99 


46.99 18.76 


4.51* 


.06 


Math 


45.36 


19.02 


47.34 20.19 


2.73 


.04 



NOTE : *** p < .001 , ** p < .01 , * p < .05. 



Discussion 



The outcome study used a pre-post matched control group design to analyze secondary data 
obtained from the Teradata warehouse maintained by the school district. A matching procedure 
was used to individually match control students to SS students on important demographic 
characteristics (ECE status, free-reduced lunch, gender, and age). Such matching lends a level of 
rigor to the evaluation of intervention effects that is seldom seen in such research, and increases 
confidence in the internal validity of the results (Cook & Campbell, 1979). 

The Evaluation Interview was used to assess the degree of knowledge and/or skills a student 
has before and after the intervention. The central measures were related to (a) empathy, (b) 
impulse control, (c) problem solving, and (d) anger management. These measures became 
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outcome criteria for establishing success of the program at the district and at the school level. As 
a District, the gains on the pretest/posttest measure were statistically significant at the .001 alpha 
level. Statistically significant gains were also noted at most of the individual schools. The results 
of this study of SS, a widely used violence prevention curriculum, provide some encouraging 
evidence of a positive effect on the central measures. Nevertheless, no statistically significant 
gains were observed on cognitive measures for the SS students. 

As in any research, there are limitations associated with this evaluation study. This primary 
outcome study had several potential limitations. First, selection criteria for participation may 
have resulted in an atypical set of schools, classrooms, and students. Second, since only the 
curriculum as a whole was evaluated, it is not possible to determine which component of it were 
responsible for the effects. This secondary outcome study was not conducted as a randomized, 
controlled trial, which limits the ability to reach firm causal conclusions about intervention 
effectiveness. Nevertheless, we partially compensated for this limitation by employing a rigorous 
computerized matching procedure to create individual student controls who were matched to 
intervention students. Thus, the study has strong internal validity, and differential group change 
may be attributed more to the interventions than to potentially confounding intervening factors 
(such as differential maturity, history, and the like discussed by Cook and Campbell, 1979). As 
in any research, these interventions may have beneficial effects on students and their families in 
ways that were not assessed in the present study, however. 

Any approach to violence prevention in the schools needs to be a multi-component and 
multi-context intervention (Stephens, 1995). An effective approach includes parents, children, 
school staff, media, police officers, local businesses, and community-based organizations. Time 
limited approaches are not effective in the long run. Approaches that focus on only one risk 
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factor (e.g., self-esteem) are also less effective. Research has shown that potentially the most 
effective programs go beyond a concentration on individual children and attempt to meaningfully 
change the climate or culture of the entire school. This is not to say that individual child-focused 
programs are ineffective and should be discontinued; they are a valuable violence prevention tool 
(Tolan et al., 1995).They do not, however, address the contextual/environmental or structural 
characteristics of a school that contribute to the incidence of violence. Programs also typically 
need to last at least 2 years before they demonstrate a change in behavior that is sustainable over 
time (Yoshikawa, 1994). As discussed above, aggressive behavior is very stable and chronic, 
making it very difficult to change with short-term, curriculum limited interventions. 

Adding violence prevention programs for long-term, systematic change, given other demands 
on schools, may be met with much resistance. In light of the many demands on them, teachers 
are often reluctant to embrace any activity that requires additional training, preparation time, or 
effort. How to address this resistance is an important issue. One strategy is to provide 
information about how violence prevention programs can actually reduce the time teachers spend 
on disciplinary problems, increasing their time for instructional activities. Another is to 
demonstrate how violence prevention efforts can reduce costs for vandalism or treatment of 
injuries related to fighting. Violence prevention programs may also increase attendance at school 
and decrease truancy, especially for children who stay home because they fear for their safety. If 
more at-risk children are actually in school, the school's ability to effect change for a child, and 
the chance that the violence prevention program will actually benefit the children most in need of 
the attention, are increased. 
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It is crucial to start anti-violence interventions early. If a child is identified as 
aggressive and at risk for academic failure at an early age, chances are that the child will 
continue to struggle over time, and the factors contributing to adjustment problems will 
grow in number, intensity, and complexity. As children mature and grow older, there is a 
shrinking window of opportunity to intervene in an effective manner. The earlier the 
intervention, the greater the chances of success. The resources (measured in time, money, 
and effort) expended by waiting until a child is in adolescence, compared to the cost of 
intervening early in a child's life, are enormous. And the pool of resources available for 
anti-violence interventions is rapidly shrinking. It is essential to continue evaluating the 
anti-violence program's effectiveness (Webster, 1993). 
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